Most Published Research Findings Are False—But a Little Replication Goes a Long Way

February 2007 | Volume 4 | Issue 2 | e28 We know there is a lot of lack of replication in research fi ndings, most notably in the fi eld of genetic associations [1–3]. For example, a survey of 600 positive associations between gene variants and common diseases showed that out of 166 reported associations studied three or more times, only six were replicated consistently [4]. Lack of replication results from a number of factors such as publication bias, selection bias, Type I errors, population stratifi cation (the mixture of individuals from heterogeneous genetic backgrounds), and lack of statistical power [5]. In a recent article in PLoS Medicine, John Ioannidis quantifi ed the theoretical basis for lack of replication by deriving the positive predictive value (PPV) of the truth of a research fi nding on the basis of a combination of factors. He showed elegantly that most claimed research fi ndings are false [6]. One of his fi ndings was that the more scientifi c teams involved in studying the subject, the less likely the research fi ndings from individual studies are to be true. The rapid early succession of contradictory conclusions is called the “Proteus phenomenon” [7]. For several independent studies of equal power, Ioannidis showed that the probability of a research fi nding being true when one or more studies fi nd statistically signifi cant results declines with increasing number of studies. As part of the scientifi c enterprise, we know that replication—the performance of another study statistically confi rming the same hypothesis—is the cornerstone of science and replication of fi ndings is very important before any causal inference can be drawn. While the importance of replication is also acknowledged by Ioannidis, he does not show how PPVs of research fi ndings increase when more studies have statistically signifi cant results. In this essay, we demonstrate the value of replication by extending Ioannidis’ analyses to calculation of the PPV when multiple studies show statistically signifi cant results. The probability that a study yields a statistically signifi cant result depends on the nature of the underlying relationship. The probability is 1 − β (one minus the Type II error rate) if the relationship is true, and α (Type I error rate) when the relationship is false, i.e., there is no relationship. Similarly, the probability that r out of n studies yield statistically signifi cant results also depends on whether the underlying relationship is true or not. Let B(p,r,n) denote the probability of obtaining at least r statistically signifi cant results out of n independent and identical studies, with p being the Most Published Research Findings Are False— But a Little Replication Goes a Long Way Ramal Moonesinghe, Muin J. Khoury, A. Cecile J. W. Janssens